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ABSTRACT 

In a study of the reliability of an English language 
proficiency test for non-native teaching assistants (NNS TAs) , 18 NNS 
TAs were administered the SPEAK Test. The results were compared with 
student evaluations of the intelligibility and clarity of the TAs 1 
classroom speech. Despite problems with the SPEAK Test, the measure 
was considered a valuable primary screening device. Several of the 
highest-scoring TAs on the SPEAK Test were not rated highly by 
students, suggesting the possibility of other variables affecting 
student evaluations. A questionnaire was administered to the students 
for evaluation of the TAs 1 communication style, and the students also 
completed the usual faculty evaluation form. The subconstruct 
"attentiveness" on the communicator style measure was the only 
variable showing a high degree of correlation with assessments of 
teacher performance. Videotapes of the TAs in a mock teaching 
situation were used to further examrne the attentiveness factor in 
the classroom performance of three or Lhe NNS TAs. It is suggested 
that teacher attentiveness can compensate for deficiency in English 
speech skills. The undergraduate student questionnaire is appended. 
fMSE) 
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In the spring of 1987, a committee was formed at Memphis 
State University to evaluate the need for special testing and 
training for non-native speaking teaching assistants (NNS TAs) . 
As the result of this committee's concerns, a research study was 
conducted in the spring of 1988. The study had two objectives — 
the reliability of the SPEAK Test for predicting the NNS TAs 1 
English speaking performance in the classroom (committee members 
were divided concerning its use) , and the emphasis that should be 
given to linguistic training if a special NNS TA training program 
were to be established. 

The first concern of the research, therefore, was to establish 
the relationship between SPEAK scores and student-rated speech 
evaluation scores. The SPEAK test had never been given at 
Memphis State University. The nineteen NNS TAs who were teaching 
classes (not directing lab sessions) during the spring semester of 
1988 were invited to participate. Eighteen agreed to be tested 
and to have their classroom performance evaluated by their 
students. 

The SPEAK test had to be conducted in two sittings because 
of scheduling conflicts, but all the NNS TAs of any one national 
origin were tested together to reduce the likelihood of 
information being passed from the first group to the second. The 
NNS TAs came from the following countries: five from India, 
three from Taiwan, two from Iran, two from Korea, and one each 
from The People's Republic of China, Egypt, France, Lebanon, 
Malaysia, and Cameroon. The audio-cassettes were sent to 
Educational Testing Services for rating. The comprehensive score 
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was used as the comparative measure for statistical analysis, A 
perfect score was 300. 

To measure the undergraduate students 1 evaluation of the NNS 
TAs 1 English speaking ability a scale, termed the Speech 
Evaluation (SE) scale, was adapted from John Orth (1982). This 
scale sought to evaluate the intelligibility and clarity of the 
NNS TAs 1 speech. Each of the items was computed by asking the 
respondents to indicate their agreement or disagreement with a 
statement about their NNS TA's speech. The five choices were 
"strongly agree, agree, neutral, disagree, or strongly disagree". 
There were eight scored items and six unscored items that were 
added as filler items to avoid response set. The item-total 
correlations ranged from 0.57 to 0.77. All scored items 
contributed to the scale's reliability. A composite score was 
computed by combining scores on the eight scored items and a 
perfect score was 40. In computing the scores, three of the 
scores had to be reversed and the neutral scores were eliminated. 
(See Appendix 1). 

The relationship between the two sets of scores was tested 
by using correlational analysis. The comprehensive ratings that 
the NNS TA received on the SPEAK test "ere compared to the mean 
ratings of the NNS TAs on the speech evaluation scale. A 
Pearson's r Product Moment Correlation Coefficient was 
calculated. The correlation between these two variables was 
0.6141 (p <001), indicating a moderate positive correlation. 

The SPEAK test did produce a varying degree of anxiety among 
the NNS TAs. Some of the NNS TAs expressed very negative 
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attitudes towards the test, and some technical problems did 
occur. Although the test did predict the top quarter and the 
bottom quarter of the sample very accurately, the grey area in 
this sample appeared to range from 180 to 260. However, given 
the problems that an institution like Memphis State has with 
funding and release time for their very limited number of 
qualified personnel, the test would have to be regarded as a 
valuable primary screening device. 

It is interesting to compare the Speech Evaluation scores of 
three NNS TAs who scored 260 on the SPEAK test. Their ratings 
among the NNS TA sample for the Speech Evaluation scale were 11th 
(26.65), 13th (23.82) and 18th (20.99) highest. Why would the 
speech of the three NNS TAs be viewed so differently by their 
undergraduate students? What variables intervene to affect the 
way an undergraduate class evaluates a NNS TA? 

This discrepancy between objective evaluations of NNS 
TAs 1 speech and student generated evaluations has been noted in 
the literature by Orth (1982) and Bailay (1982). In both cases 
the researchers surmised that some variable or variables 
intervene to affect the undergraduates 1 evaluations. To take 
this assumption one step further, Bailey attempted to determine 
whether something more than the NNS TAs 1 English speaking ability 
affects their overall evaluations as an effective teacher. 

Bailey (1982) used a coding scheme to determine what 
communication moves were positively related to a high teaching 
evaluation. What she discovered was that bonding (any act that 
helps the TA establish a bond, such as apologizing or use of 
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humor) and eliciting (any act where the TA invites comments or 
questions) were positively related for native speaking TAs, and 
inform moves (any act where information is given) were negatively 
related for non-native speaking TAs, 

Bailey (1982) then developed typologies of TAs who had 
varying degrees of acceptance by their undergraduate students. 
The least successful TAs had little interaction with their 
students and were termed either Active Unintelligibles or 
Mechanical Problem Solvers. However, the three remaining groups 
(Knowledgeable Helpers/Casual Friends, Entertaining Allies, and 
Inspiring Cheerleaders) displayed characteristics that made them 
effective teachers according to their student-rated evaluations. 
What distinguished these latter three typologies were their 
friendliness, their relaxed styles, their degree of involvement 
with their students, their liveliness, and their ability to play 
to their students as to an audience* 

The Memphis State research attempted to tap these same 
characteristics but in a different way. Using a quest ionnaj.re, 
the Communicator Style Measure, developed by R. Norton (1977) for 
the field of instructional communication, the undergraduate 
students were asked to evaluate the communicator style of their 
NNS TAs. From 11 subconstructs 5 were chosen that best 
approximated the characteristics exhibited by Bailey's (1982) top 
three typologies: friendly , relaxed , attentive, animated , and 
dramatic . 

The Communicator Style Measure questionnaires were 
distributed at the same time as the Speech Evaluation scale 
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(Orth, 1982), toward the end of the semester when the 
undergraduates had become sufficiently familiar with the NNS TAs 1 
speech and communicator style. The variables were computed by 
asking the undergraduates to indicate their agreement or 
disagreement with a statement about the NNS TA ! s communicator 
style on a five point scale of "strongly agree, agree, neutral, 
disagree, strongly disagree. 11 Norton reports reliabilities for 
the communicator style item measures utilized in this study as 
follows: friendly (.37), animated (.56), attentive (.57), 
dramatic (.68), and relaxed (.71). There were four scored items 
for each variable and only these items were included. The order 
was mixed to counter response set. Two item scores were 
reversed. Each composite score was computed by combining the 
four scored items and a perfect score was 20. {See Appendix 2). 

To evaluate the NNS TA's effectiveness as a teacher the 
undergraduates participated in the regular faculty evaluation 
process using the SIRS instument presently in use at Memphis State 
University. This form was adapted by Arreola (1973) from the 
Michigan State University SIRS evaluation form developed by 
Centra in 1969. This measure constituted the dependent variable 
for the research analysis. 

Two factors normally included in this evaluation form — 
course demands and course organisation —were excluded from the 
analysis because TAs do not have control over these factors. 
Internal consistency and reliability for the three remaining 

factors are as follows: 1. Instructor Involvement .81 

2. Student Interest .79 

3 . Student-Instructor Interaction . 84 



The variables were computed by asking the undergraduates to 
respond to statements about the NNS TA's teaching by indicating 
their agreement or disagreement on a five-point scale (strongly 
agree, agree, neutral, disagree, strongly disagree). There were 
four scored items for each of the above measures. Two single 
item statements to evaluate teaching competence and teaching 
effectiveness were also included in the analysis. A single 
composite score was computed by combining the four measures and 
the twc single item responses and a perfect score was 90. 

A stepwise multiple regression was computed using six 
independent variables — the five communicator style subconstructs 
and the speech evaluation score — and the teaching evaluation 
measure (SIRS) as the dependent variable. The teacher's 
performance within a class was used as the unit of analysis. 

The subconstruct attentive was the only variable to enter 
the equation. It was highly significant with a correlation of 
0.85 (p <.001) and it explained 73 percent of the variance. No 
further variables were entered into the equation because the 
remaining five variables had nonsignificant t values and would 
therefore not add to the predictive value of the equation. 

Norton (1983) describes attentive as signaling an ongoing 
willingness to provide feedback that the person's messages are 
being processed in an alert and/or understanding manner. This 
description clearly would embrace the bonding and eliciting that 
Bailey (1982) found to be positively related to effective 
teaching . 
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There were three NNS TAs in the study whose demonstration 
of, or lack of, attentiveness was particularly significant • 
Videotapes were made of these three NNS TAs to see if the NNS TAs 
would demonstrate attentiveness in a mock teaching situation. 
All three of these NNS TAs were invited to present a ten 
minute lecture on any topic that they were comfortable with to a 
a master's level TESOL assessments class. This class was chosen 
in order to keep the choice of NNS TAs and the video-taping 
confidential. Interestingly, communicator style preference was 
evident in the mock teaching presentation • 

The first NNS TA, from India, scored 270 on the SPEAK 
test, was given the highest score in the sample on the student- 
rated speech evaluation (32.43 out of 40), aad the third highest 
attentiveness rating (15.33 out of 20), and the highest teaching 
evaluation (75.65 out of 90). He taught mathematics. 

The videotape demonstrated that he made excellent use of 
inclusive pronouns to build a sense of commonality in the 
classroom. Though his speech was deliberate at times, he spoke 
clearly, reinforcing what he was writing on the board, and 
elaborating effectively at key juncture points in the problem. 
His game (demonstrated in the videotape) was designed to build 
confidence among the students, and he acted as a cheerleader 
extolling the students to enjoy the game and assuring them of how 
easy mathematics really is. 

The second NNS TA, from the Republic of China, Taiwan, had 
spoken English for the shortest period of time among the NNS TAs. 
She scored 230 on the SPEAK test and her student-rated speech 
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evaluation was in the middle range of the sample (27,27 out of 
40), She shared with another NNS TA the highest rating in 
attentiveness (15.42 out of 20), and received the second highest 
teaching evaluation (74,59 out of 90), She also taught 
mathematics. 

This NNS TA, more than any other, demonstrated how 
effectively an attentive attitude compensates for poor English 
speaking skills. She honestly admitted her nervousness, showed 
respect for others 1 opinions and asked the students to play with 
a math formula that was individualized, making everyone feel 
important and involved. She also used inclusive pronouns and 
gestures to build commonality in the classroom. 

The third NWS TA, from Cameroon, scored 260 on the SPEAK 
test, but received the lowest score in the sample on the student- 
rated speech evaluation (20.99 out of 40). He had the lowest 
attentiveness rating (11.93 out of 20), and the lowest teaching 
evaluation (53.00 out of 90). He taught accounting. 

This NNS TA lectured throughout his presentation exactly as 
he did in the two minute segment on videotape, following a well- 
organized, pre-arranged lesson plan. There was no interaction 
with the students and little board work. Therefore, the students 
had no opportunity to see any words he was mispronouncing. He 
moved very little from behind the desk and his humor was directed 
at the students rather than shared with them. 

It was interesting that The two highly attentive NNS TAs 
both chose to accommodate their audience ? s non-math background by 
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introducing a game that would simplify mathematics for the 
students and create a relaxed atmosphere for learning. They both 
expressed their enthusiasm for their chosen discipline but 
allowed that their undergraduate students usually did not share 
this enthusiasm and that they understood that others had a right 
to different opinions. Based upon this honest recognition of 
differing attitudes they set out to win over what they expected 
were reluctant mathematicians. The TESOL students asked 
questions and became so involved in the presentations that both 
original versions of the videotapes ran about 30 minutes. There 
was a lot of the Entertaining Ally and Inspiring Cheerleader 
(Bailey, 1982) in both these presentations. 

The third NNS TA, on the other hand, never attempted to 
accommodate nis audience. He shared with the class his general 
disdain for undergraduate accounting students and indicated that 
his role was to show them how poorly high school bookkeeping had 
prepared them for college level accounting. His attitude toward 
his discipline was equally negative. There was no interaction 
with the class. 

Although this research is limited by its size — only 18 
NNS TAs participated — and its single location — Mid-South 
undergraduates may be unique in the value they place on a high 
degree of support and therefore on an attentive style from their 
TAs — it does raise some issues that deserve further study. It 
suggests that poor English speaking skills can be compensated for 
by expressing an attentive style of communication. It reinforces 
earlier research findings (Orth, 1982, Bailey, 1982) that speech 
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alone cannot predict effective teaching performance in the 
classroom . Furthermore, it raises questions about the current 
method of choosing TAs that emphasizes their academic ability 
rather than their interpersonal skills. 
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Appendix 1 - Speech Evaluation Scale - copyright John Orth, 1982 



Undergraduate Questionnaire 2 - SB-Scale 

The following statements express opinions that students might have 
about an instructor! There are no right or wrong answers. Many 
students might agree with each statement and many others might 
disagree. We are interested in your honest fueling about whether each 
statement applies to your instructor in this class. Please work 
quickly but neatly. Do not spend too much time thinking about each 
statement, but give your first reaction, marking it on this sheet 
using the following scale. 

YES! « STRONGLY AGREE 

yes ■ AGREE 

7 m NEUTRAL or UNDECIDED 

no - DISAGREE 

NO! - STRONGLY DISAGREE 



1. I have to listen very closely to understand what the instructor 
says. 

YES! yea 7 no NO! 

2. The instructor is able to communicate clearly and effectively in 
English. 

YES! yes 7 no NO! 

3. I find it hard to concentrate on what the instructor says because 
of the way s/he speaks. 

YES! yes 7 no NO! 

4. The instructor speaks clearly and is easy to understand. 

YES! yss ? no NO! 

5. Speaking English seems to be difficult for the instructor. 

YES! Y«o 7 no NO! 

6. I don't have any problems understanding "the instructor. 

YES! yea 7 no NO! 

7. The instructor sounds confident. 

YES! yes 7 no NO! 

8. The instructor has difficulty expressing her/himself clearly and 
effectively in English. 

YES! yes ? no NO! 



ERLC 



13 



12 



9. The instructor has a heavy foreign accent. 

YES! yes ? no NO! 

10. The instructor's speech is annoying and distracting. 

YESl yes ? no NO! 

11. I have difficulty understanding the instructor, 

YES! yes ? no NO! 

12. The instructor's English speaking ability is limited. 

YES! yes ? no HO! 

13. The instructor can barely Bake her/himself understood. 

YES! . yes ? no HO! 

14. The instr ctor sounds unsure of her/himself. 

YES! yes ? no HOI 
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Appendix 2 - Communicator Style Measure - Copyright Robert 

Norton! 1977 



Undergraduate Questionnaire 3 - Communicator Style Measure 

The following statements express opinions regarding your instructor. We 
are interested in your honest reactions to each statement. Please work 
quickly but neatly, giving your first reaction, without spending too 
much time thinking about each statement, Mark your answers on this 
sheet using the following scale: 

YES I « STRONGLY AGREE 
ye3 » AGREE 

? » NEUTRAL or UNDECIDED 
no « DISAGREE 
NO! « STRONGLY DISAGREE 



1. s/he readily expresses admiration for others, 

YES! yes ? no NO! 

2. To be friendly, s/he habitually acknowledges verbally other's 
contributions. 

YES! yes ? no NO! 

3. S/he has some nervous mannerisms in her/his speech. 

YES! yes ? no NO! 

4. S/he is a very relaxed communicator. 

YES! yes ? no NO! 

5. S/he can always repeat back to a person exactly what 
was meant. 

YES I yes ? no NO! 

6. The rhythm or flow of her/his speech is sometimes affected 
by her/his nervousness. 

YES! yes ? no NO! 

7. Under pressure s/he comes across as a relaxed speaker. 

YES! yes ? no NO! 



8. Her/his eyes reflect exactly what s/he is feeling when 
s/he communicates. 

YES! ye3 ? no NO! 
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9. S/he dramatizes a lot, 

YES! yes ? no NO! 

10. Usually, s/he deliberately react* in such a way that people 
know that s/he lit listening to them, 

YES I yes ? no NO! 

11. Regularly s/he 'slls jokes, anecdotes and stories when 
s/he communicates. 

YES t yes ? no NO! 

12. S/he tends to constantly gesture when s/he communicates. 

YES I" yes ? no NO* 

13. Often s/he physically and vocally acts out what s/he wants 
to communicate. 

YES! yes ? no NO! 

14. S/he is always an extremely friendly communicator. 

YES! yes ? no NO! 

15. S/he really likes to listen very carefully to people. 

YES! yes ? no NO! 

16. S/he is very exspressive nonverbal ly in social situations. 

YES! yes ? no NO! 

17. Whenever c/he communicates, s/he tends to be very 
encouraging to people. 

YES! yes ? no NO! 

18. S/he actively uses a lot of facial expressions when s/he 
communicates, 

YES! yes ? no NO! 

19. S/he very frequently exaggerates to emphasize a point. 

YES! yes ? no NO! 



20. S/he is an extremely attentive communicator. 
YES! yes ? no NO! 



*6 



INSTRUCTOR 



DEPT. 



COURSE NO. 



SECTION 



DIRECTIONS 

THIS FORM ENABLES YOU TO RATS BOTH THE INSTRUCTOR AND THE 
COURSE ON SEVERAL CHARACTERISTICS. PLEASE RESPOND AS 
ACCURATELY AND HONESTLY AS YOU CAN. LEAVE BLANK ANY ITEM 
THAT DOES NOT APPLY. A FREE RESPONSE SECTION IS PROVIDED ON 
THE REVERSE SIDE. 

USE ASOFT LEAD PENCIL TO RESPOND TO EACH ITEM ACCORDING TO 
KEY PRINTED AT RIGHT. DO NOT USE INK OR BALLPOINT PENS. MAKE 
HEAVY BLACK MARKS THAT FILL THE CIRCLE COMPLETELY. ERASE 
CLEANLY ANY ANSWER YOU WIS; I TO CHANGE. 
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SA - If you strongly agree with the statement 
A - If you agree with the statement 
N — If you neither agree nor disagree 
D — If you disagree with the statement 

SD - If you strongly disagree with the statement 



WRONG 
WRONG 
WRONG 
RIGHT 



■ 1. 
» 2. 
i 3. 

■ 4. 

■ 5. 

■ 6. 
i 7. 
i 8. 
• 9. 
» 10. 
■11. 
' 12. 
' 13. 
' 14. 

1 15. 

' 16. 
' 17. 
' 18. 
' 19. 
' 20. 
'21. 
'22. 

'23. 
' 24. 
' 25. 
'26. 

27. 
28. 
29. 
30. 
31. 
32. 



Tne instructor was enthusiastic when presenting course material 1, 

The instructor seemed to be interested in teaching 2. 

Th » instructor* s use of examples or personal experiences helped to get points across in class 3. 

The instructor seemed to be concerned with whether the students learned the material 4, 

You were interested in learning the course material 5, 

You were generally attentive in class 6. 

You felt this course challenged you intellectually 7. 

You have become mc e competent in this area due to this course 8. 

The instructor encouraged students to express opinions 9. 

The instructor appeared receptive to new ideas and others' viewpoints 10. 

The student had an opportunity to ask questions 11 

The instructor generally stimulated class discussion 12 

The instructor attempted to cover too much material 13 

The instructor generally presented the material too rapidly 14. 

The homework assignments were too time consuming relative to their contribution to your 
understanding of the course material 15. 

You generally found the coverage of topics in the assigned readings too difficult 16. 

The instructor appeared to relate the course concepts in a systematic manner 17. 

The course was well organized 18. 

The course materials appeared to be presented in logical content units 19. 

The direction of the course was adequately outlined 20. 

This course made a significant contribution to your overall personal educational objectives 21. 

What percentage of the course material covered do you feel you actually learned? 

(a) more than 90% (b) about 80% (c) about 70% (d) about 60% (e) less than 60% 22. 

The instructor adequately assessed how well students mastered the course objectives 23. 

The stated course objectives were reflected in the exams 24. 

The instructor appeared to be thoroughly competent in his area 25. 

In general, the instructor was an effective teacher 26. 
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STUDENT BACKGROUND: Select the most appropriate alternative. YES 

Do you have confidence that these ratings will be taken seriously? 27. ® 

Was this form administered fairly and correctly? 28. © 

Was this a required course for you? 29. © 

Are you a major in the area in which this course is being taught? 30. © 

What grade do you expect to receive in this course? 31.0 

What is your overall GPA? (a) 2.2 or less (b) 2.3-2.5 (c) 2.6-2.9 (d) 3.0-3.3 (e) 3.4-4.0 32.© 



NO 
® 
® 



© 
© 



© ® © 
0 © © 



OPTIONAL ITEMS: Items 1 through 30 below may be usea to respond to items specified by the instructor. 



1®®©®© 
2®®©®® 
3®®©®© 
4®®©®© 
5®®©@© 
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7®®©®© 
8®®©®© 
9®®©®® 
10®®©®® 



11®®©®© 
12®®©®© 
13®®©®® 
14®®©®® 
15®®©®® 



16®®©®® 
17®®©®® 
13®®©®© 
19®®©®© 
20®®©®© 



21®®©®© 
22®©©®® 
23®©©®® 
24®®©®® 
25®®©®© 



26®®©@© 
27®®©®® 
28®©©®© 
29®®©®© 
30O©©®© 



SEE REVERSE SIDE FOR FREE RESPONSE SECTION 



